Information-Theoretical Criteria for Characterizing the Earliness of Time-Series Data

Abstract

Biomedical signals constitute time-series that sustain machine learning techniques to achieve classification. These signals are complex with measurements of several features over, eventually, an extended period. Characterizing whether the data can anticipate prediction is an essential task in time-series mining. The ability to obtain information in advance by having early knowledge about a specific event may be of great utility in many areas. Early classification arises as an extension of the time-series classification problem, given the need to obtain a reliable prediction as soon as possible. In this work, we propose an information-theoretic method, named Multivariate Correlations for Early Classification (MCEC), to characterize the early classification opportunity of a time-series. Experimental validation is performed on synthetic and benchmark data, confirming the ability of the MCEC algorithm to perform a trade-off between accuracy and earliness in a wide-spectrum of time-series data, such as those collected from sensors, images, spectrographs, and electrocardiograms.

1. Introduction

A time-series (TS) consists of measurements or observations acquired and organized sequentially over time. In this context, one or multiple variables may be examined, being the first-named univariate time-series (UTS) and the second multivariate time-series (MTS). Several data mining application areas deliver this sort of data, such as medicine, economy, meteorology, and marketing. Standard TS classification involves using temporal data for constructing a classifier, which can predict the class label of a new given TS, with satisfactory accuracy.

Early classification (EC) is an extension of the TS classification problem, and it arises in scenarios where the anticipation of the prediction is beneficial. This matter has recently been a relevant subject of study, due to its several time-sensitive applications. For instance, a medical study [1] described how clinical data revealed that infants diagnosed with sepsis disease suffered from an unusual heartbeat twenty-four hours before the diagnosis. In this case, supervising the TS data of the infant’s heartbeat and being able to classify it in advance may lead to effective early diagnosis and treatment.

The work from Xing et al. [2] was one of the first to formulate the problem of EC, proposing to unveil a timestamp from which the information of the TS from that point on is irrelevant. As stated by Xing et al. [3], it is vital to distinguish EC from classic TS prediction, where the goal is to forecast values given the whole TS. EC of temporal data consists of anticipating the classification by using only a portion of the available information, without compromising the prediction quality. There are two essential requirements for an early classifier: being able to designate the earliest time location (timestamp) of accurate classification, and ensuring an accuracy close to the case of using the full-length TS.

Several methods addressing the early classification problem have been proposed in the last years [2,4,5,6,7,8]. These approaches require a learning stage followed by a classification step, being able to assign a class label to a single incomplete time-series. Notwithstanding the significant advances in EC, there are temporal datasets for which it is intrinsically hard to perform early predictions, due to phenomenons like high conductance of the underlying Markov model, or cumulative errors throughout the time measurements.

In contrast, we propose a dataset investigation, where the information from the entire collection of time-series (and not only a single TS), as well as their respective class labels, is the subject of study. Our approach explores the knowledge contained in the data via the temporal correlation of the variables. In general, testing all possible timestamps without external information would require an expensive, and in some cases, computationally unfeasible, cross-validation search. We propose to use information-theoretical criteria to perform this analysis efficiently throughout the whole data.

Overall, the main contribution of this work is applying information-theoretic criteria for examining the EC opportunity in a dataset containing univariate or multivariate TS, extracting a global timestamp (for the overall data) from which accurate prediction might be performed. An EC method, like those mentioned before, can then be applied, with a priori knowledge of the EC opportunity in the data.

We assessed the merits of the proposed method in a wide-spectrum of time-series data, such as those collected from sensors, images, spectrographs, and electrocardiograms. In particular, we found that for a collection of time-series that traces the electrical activity recorded during one heartbeat, the first 30% of the signal contains enough information to discriminate between a normal heartbeat and a myocardial infarction, leveraging the development of novel methods to focus in this fraction of the signal. An implementation of the proposed criteria, through a procedure called Multivariate Correlations for Early Classification (MCEC), is made freely available to the comunity at https://joaopbeirao.github.io/MCECalgorithm/.

The paper is organized as follows. In Section 2, we review some basic concepts of Bayesian networks and multivariate correlation. In Section 3, we present our EC opportunity algorithm, followed by experimental results over benchmark data; results in synthetic data are left for the Appendices. Finally, we draw some conclusions and discuss future work.

2. Background

We start by introducing Bayesian networks as they have a well-established framework to measure model complexity. In this regard, we choose two criteria: minimum description length and Akaike information criterion. Finally, we present multivariate correlations to assess data earliness by deriving the dependencies among the variables through a concrete Bayesian network calibrated with the adopted criteria.

2.1. Bayesian Networks

Probabilistic graphical models attempt to describe the behaviour of complex systems using a graph-based framework for representing the probability distributions [9]. Bayesian networks (BNs) are probabilistic graphical models for describing complex domains, and they can be used to represent the information about an uncertain system [10,11,12]. The BN representation consists of a directed acyclic graph G, characterized by a set of nodes $\mathcal{N}=\left\{X_1,X_2,\dots ,X_n\right\}$ and a set of directed edges E. Considering a $G=(\mathcal{N},E)$, each node (vertex) corresponds to a random variable $X_i$, and the edges (arrows), that connect the nodes in a specific direction, describe the probabilistic dependencies between the random variables. For each node $X_i$, two sets can be defined: the set of parents ${\Pi }_{X_i}$ and the set of non-descendants ${\Phi }_{X_i}$. The structure of a BN is based on the assumption that each node $X_i$ is conditionally independent of ${\Phi }_{X_i}$, provided that the values of the variable in ${\Pi }_{X_i}$ are known. The group of local probability models, representing the dependence of each variable $X_i$ on ${\Pi }_{X_i}$, specifies the parameters for describing the network structure. These form the set of conditional probability distributions

$$\mathsf{\Theta }={\left\{{\theta }_{X_i|{\Pi }_{X_i}}\right\}}_{i\in \left\{1,\dots ,n\right\}},$$

where ${\theta }_{X_i|{\Pi }_{X_i}}=P(X_i=x_i\mid {\Pi }_{X_i}={\omega }_i)$, associated to each node $X_i$ and conditioned on ${\Pi }_{X_i}$.

A BN $\mathcal{B}=(G,\mathsf{\Theta })$ is comprised of the direct acyclic graph structure G together with the set of parameters $\mathsf{\Theta }$. The joint probability distribution defined by this representation is calculated as:

$$P_{\mathcal{B}}(X_1,\dots ,X_n)=\prod _{i=1}^n{P}_{\mathcal{B}}(X_i\mid {\Pi }_{X_i})=\prod _{i=1}^n{\theta }_{X_i|{\Pi }_{X_i}}.$$

(1)

For a given a multinomial dataset D of size N, the problem of learning a BN consists of designing the $\mathcal{B}=(G,\mathsf{\Theta })$ that best represents D, according to some scoring function [13]. This scoring criterion corresponds to the search guide for evaluating the effectiveness of the network in representing the data. Moreover, when the structure of the network is fixed, the parameters $\mathsf{\Theta }$ that optimize the likelihood, for a given dataset, are those described by the observed frequency estimates:

$${\widehat{P}}_{\mathcal{B}}(X_i=x_i\mid {\Pi }_{X_i}={\omega }_i)=\frac{|D_{x_i,{\omega }_i}|}{|D_{{\omega }_i}|},$$

(2)

for which $|D_{x_i,{\omega }_i}|$ represents the number of instances in D, where $X_i$ takes the value $x_i$, and its parents ${\Pi }_{X_i}$ take the value ${\omega }_i$. Similarly, $|D_{{\omega }_i}|$ denotes the number of instances in D, where ${\Pi }_{X_i}$ takes the value ${\omega }_i$.

The Minimum Description Length (MDL) principle is known as an Occam’s razor approach to select, for a given dataset, the best fitting model and its parameters [14]. It is a widely used metric which states that, for a certain data and a number of alternative models, the best option corresponds to the simplest model [15,16,17]. In the problem of learning a BN, the Bayesian Information Criterion (BIC) is known as the MDL score. It is concerned with analysing the trade-off between the Log-Likelihood (LL) of the dataset D (the effectiveness of the fit to the data) and the complexity of the model. This scoring function [13] is defined as

$$MDL(D\mid \mathcal{B})=-LL(D\mid \mathcal{B})+\frac{{log}_{2}N}{2}|\mathcal{B}|,$$

(3)

where N corresponds to the size of the data, and $|\mathcal{B}|$ represents the model dimension (number of parameters in $\mathcal{B}$). The LL term quantifies the amount of information required to describe the dataset D, using $\mathcal{B}$. Conversely, the penalty term measures the amount of information needed to encode the model $\mathcal{B}$. The overall goal of the MDL score is to elicit the model that most effectively fits the dataset, provided that its complexity is as low as possible, in this way avoiding overfitting.

Similarly to the MDL scoring function, the Akaike Information Criterion (AIC) [18] corresponds to a measure of the quality of statistical models for describing a given dataset. In the problem of learning a BN, the difference between MDL and AIC is associated to the penalty applied to the number of parameters $|\mathcal{B}|$. The AIC scoring function [13] can be defined as:

$$AIC(D\mid \mathcal{B})=-LL(D\mid \mathcal{B})+|\mathcal{B}|.$$

(4)

In Equation (3), the second term quantifies the amount of information required to encode the model $\mathcal{B}$, where each parameter in the set $\mathsf{\Theta }$ is considered to use $\frac{1}{2}{log}_{2}N$ bits. Conversely, in Equation (4) each parameter of $\mathsf{\Theta }$ is considered to use 1 bit. This means that the penalization on the number of independent parameters is stronger in the MDL scoring function than in the AIC score. Likewise for the MDL score, the best model corresponds to the one that minimizes Equation (4).

Literature complies with the fact that these two criteria demonstrate different properties for model selection and that they are appropriate according to specific conditions [19,20,21]. According to Vrieze [21], MDL is considered to be consistent in selecting the true model, with probability close to one, given that the true model is in the set of candidate models. On the other hand, if the true model is not in the set of alternatives, AIC is considered to be effective, since it selects the model that minimizes the mean squared error of the estimation. However, both criteria are unsuitable for dealing with low dimensional datasets for which the number of instances is close to the number of parameters to estimate [22].

2.2. Multivariate Correlations

We briefly introduce multivariate correlations to derive a concrete Bayesian network from which we deduce model complexity. From a statistical point of view, the concept of correlation between variables attempts to measure the relationships and dependencies among them. The knowledge of how the variables are related, as well as of what inferences can be made about their causal relationships, is useful for drawing conclusions about potential predictive relationships to be analyzed and exploited.

For a finite set of discrete random variables $S={\left\{X_i\right\}}_{i=1,\dots ,n}$, with a joint probability distribution $P_S(X_1,\dots ,X_n)$, the total correlations between those variables can be defined as [23]:

$$I(X_1,\dots ,X_n)=\sum _{i=1}^{n}H\left(X_i\right)-H(X_1,\dots ,X_n).$$

(5)

In this case, the mutual information measures the dependencies among the variables, i.e., the amount of information that these quantities give about each other.

Let a structural relation R be a subset of the system S. Its joint probability distribution corresponds to the marginal distribution from S

$$P_R(X_{R_1},\dots ,X_{R_k})=\sum _{X_i\notin R}{P}_S(X_1,\dots ,X_n),$$

(6)

where k is the number of elements in R.

Following the maximum entropy principle applied to fixed marginals [24], we say that a structure associated to the system S with underlying joint probability $P_S$ is a pair $\left(\mathcal{S},P_{\mathcal{S}}\right)$, where $\mathcal{S}={\left\{R_j\right\}}_{j=1,\dots ,k}$ is a collection of structural relations and $P_{\mathcal{S}}$ is (another) joint probability distribution over S, such that:

• No $R_i\in \mathcal{S}$ is contained in another $R_j\in S$ (or in other words, ${\forall }_{i\ne j}{R}_i⊈R_j$);
• Every $X_i\in S$ is included in at least one $R_j\in \mathcal{S}$;
• $P_{\mathcal{S}}$ is the solution to the optimization problem:

  $$\underset{P\in \mathcal{P}}{\max }H\left(P\right)\mathrm{s}.\mathrm{t}.\sum _{X_i\notin R_j}{P}_{\mathcal{S}}(X_1,\dots ,X_n)=\sum _{X_i\notin R_j}{P}_S(X_1,\dots ,X_n),\forall R_j\in \mathcal{S},$$

  where $\mathcal{P}$ is the set of probability distributions of the variables from S.

For example, from the set of discrete random variables $S=\left\{X_1,X_2,X_3,X_4\right\}$, some admissible structures $\mathcal{S}$ are $\left\{\left\{X_1,X_2,X_3\right\},\left\{X_4\right\}\right\}$, $\left\{\left\{X_1,X_2\right\},\left\{X_3,X_4\right\}\right\}$ or $\left\{\left\{X_1,X_2\right\},\left\{X_1,X_4\right\},\left\{X_2,X_3,X_4\right\}\right\}$. On the other hand, the structure

$$\mathcal{S}=\left\{\left\{X_1,X_2,X_3\right\},\left\{X_1,X_3\right\},\left\{X_4\right\}\right\}$$

is not an acceptable structure since the relation between $X_1$ and $X_3$ is included in two structural relations, which violates the first condition. Similarly, $\mathcal{S}=\left\{\left\{X_1,X_2\right\},\left\{X_2,X_4\right\}\right\}$ does not consist of a proper structure because the variable $X_3\in S$ is not part of any structural relation from $\mathcal{S}$, as required by the second condition.

For a given system $S={\left\{X_i\right\}}_{i=1,\dots ,n}$ and an associated set of structural relations $\mathcal{S}={\left\{R_j\right\}}_{j=1,\dots ,k}$, the mutual information $I\left(S\right)$ represents the maximum amount of information that the variables $X_i$ from S provide about each other. On the other hand, $I\left(\mathcal{S}\right)$ quantifies the information described by the correlations only inside the structural relations $R_j$. The difference $I\left(S\right)-I\left(\mathcal{S}\right)$ measures the knowledge of the dependencies and relationships between the variables of S that are not included in the relations that compose $\mathcal{S}$. From Equation (5), this value can be described as a difference of entropies:

$$\begin{array}{c}\hfill I\left(S\right)-I\left(\mathcal{S}\right)={\displaystyle \sum _{i=1}^n}H\left(X_i\right)-H\left(S\right)-{\displaystyle \sum _{i=1}^n}H\left(X_i\right)-H\left(\mathcal{S}\right)=H\left(\mathcal{S}\right)-H\left(S\right).\end{array}$$

(7)

The above expression is always non-negative because the distribution of $\mathcal{S}$ has the maximum entropy in a set where the distribution of S belongs. Seeing that the entropy quantifies the average uncertainty of a random variable, $H\left(\mathcal{S}\right)-H\left(S\right)$ represents the information given by the existing correlations in S, that is not incorporated in the structural relations from $\mathcal{S}$.

3. Proposed Method

We now set out to derive the proposed EC method for temporal data. For convenience, we start by introducing a few additional notations. Let a dataset D be a collection of pairs $(T_j,c_j)$ for all $j\in \left\{1,\dots ,N\right\}$, where $T_j$ consists of a TS, $c_j$ corresponds to its respective class label and N is the number of instances in D. In general, a TS is defined as a vector of length L

$$T_j=\left({\mathbf{x}}_1^{\left(j\right)},{\mathbf{x}}_2^{\left(j\right)},\dots ,{\mathbf{x}}_L^{\left(j\right)}\right),$$

(8)

where each component ${\mathbf{x}}_k^{\left(j\right)}=\left(x_{k1}^{\left(j\right)},x_{k2}^{\left(j\right)},\dots ,x_{km}^{\left(j\right)}\right)$ consists of m features measured at time point (TP) $k\in \{1,\dots ,L\}$. The object of TS classification is to associate a class label c to a given time-series T (not necessarily in the data).

Consider a TS T, as in Equation (8), representing the evolution of the random vector $\mathbf{X}=(X_1,\dots X_m)$ over time, and its respective class label, which is denoted by the random variable C. The set of ${\mathbf{X}}_k$ can be viewed as a collection of time-dependent discrete random variables, for which a joint probability distribution can be defined. Note that, since a TS is chronologically organized, it is relevant to analyze the dependence of variables on their early states, i.e., the degree of dependence of $\mathbf{X}$ at a certain TP on the value observed at a previous instant. Similarly, the correlation between C and ${\mathbf{X}}_k$ quantifies the dependence that the vector $\mathbf{X}$ at TP k has on the class label. In the EC context, the focus is to study systems where the class labels verify a high dependence on a certain amount of early states of ${\mathbf{X}}_k$, while the remaining TPs are dispensable for a satisfactory classification.

Consider the finite set of discrete random variables S to be composed of the TS T together with its respective class label C. The system

$$S=\left\{{\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_n,{\mathbf{X}}_{n+1},\dots ,{\mathbf{X}}_L,C\right\}$$

has an associated joint probability distribution $P_S({\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_L,C)$, where L represents the TS length. The goal is to find the value n and the distribution $P_S^n$, such that $P_S^n(C\mid {\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_n)\approx P_S(C\mid {\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_L)$. Therefore, $P_S^n(C\mid {\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_n)$ and $P_S(C\mid {\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_L)$ describe the probability of the class label C occurring, provided that the first n, or all variables of T, are known, respectively.

In some cases, the joint probability distribution $P_S$ is not known in advance; thus, it has to be computed from the data, through maximum likelihood estimation. In particular, for a dataset D of size N, the distribution $P_S$ that maximizes the likelihood of D is such that

$${\widehat{P}}_S({\mathbf{X}}_1={\mathbf{x}}_1,\dots {\mathbf{X}}_L={\mathbf{x}}_L,C=c)=\frac{|D_{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_L,c}|}{N},$$

(9)

where $|D_{{\mathbf{x}}_1,\dots ,{\mathbf{x}}_L,c}|$ is the number of instances in D for which ${\mathbf{X}}_i$ takes the value ${\mathbf{x}}_i$ and C the value c.

Given the system S, the set of structural relations, defined by

$${\mathcal{S}}_n=\left\{\left\{{\mathbf{X}}_1,\dots ,{\mathbf{X}}_n,{\mathbf{X}}_{n+1},\dots ,{\mathbf{X}}_L\right\},\left\{{\mathbf{X}}_1,\dots ,{\mathbf{X}}_n,C\right\}\right\},$$

depends on the value of n and it corresponds to a structure that respects the previously described properties. Considering the sets $A_n=\left\{{\mathbf{X}}_1,\dots ,{\mathbf{X}}_n\right\}$ and $B_n=\left\{{\mathbf{X}}_{n+1},\dots ,{\mathbf{X}}_L\right\}$, the structure is represented as ${\mathcal{S}}_n=\left\{\left\{A_n,B_n\right\},\left\{A_n,C\right\}\right\}$. The structural relation $A_n$ contains information about the evolution of the variable $\mathbf{X}$ until the TP n, i.e., the early stages of the collection of TS. On the other hand, $B_n$ describes the remaining instants of $T_i$, which can be viewed as the knowledge about the later stages of the variable $\mathbf{X}$. Finally, C represents the class label information from the collection of TS. The structure ${\mathcal{S}}_n$ can be seen as a simplified model of the system S. It is expected to include the correlations between the early and the later information about the TS ($A_n$ and $B_n$), as well as between the early states of $T_i$ and the knowledge about their classes ($A_n$ and C). Conversely, the correlations between $B_n$ and C are not preserved because the idea is to study the possibility of describing the class from the early states $A_n$, while neglecting the information from $B_n$. The probability distribution of ${\mathcal{S}}_n$ is obtained based on Theorem 1 (the proof can be found in Appendix A) and considering the BN represented in Figure 1.

Theorem 1.

Consider the Bayesian network${\mathcal{B}}_n=(G_n,{\mathsf{\Theta }}_n)$with$G_n$given by Figure 1 and${\mathsf{\Theta }}_n$calculated according to Equation (2). The structure $({\mathcal{S}}_n,P_{{\mathcal{S}}_n})$ over S has a probability distribution equal to the joint probability distribution of ${\mathcal{B}}_n$, that is, $P_{{\mathcal{S}}_n}=P_{{\mathcal{B}}_n}$.

Given the structure of the Bayesian network from Figure 1, we have ${\Pi }_{A_n}=\varnothing$, ${\Pi }_{B_n}=\left\{A_n\right\}$ and ${\Pi }_C=\left\{A_n\right\}$, and by Equation (1) we have:

$$P_{{\mathcal{S}}_n}=P\left(A_n\right)P(B_n\mid A_n)P(C\mid A_n).$$

(10)

where each (conditional) probability is obtained by the observed frequencies given by Equation (2). From Equation (7) and for each value of n, the difference of entropy applied to these context is represented as:

$$I\left(S\right)-I\left({\mathcal{S}}_n\right)=H\left({\mathcal{S}}_n\right)-H\left(S\right).$$

(11)

As stated above, we want to measure the loss of information about class when we lose correlations. This can be performed using conditional entropy, namely

$$H_{P_{{\mathcal{S}}_n}}(C\mid X_1\dots X_L)-H_{P_S}(C\mid X_1\dots X_L)=H(C\mid A_n)-H(C\mid A,B),$$

(12)

where $A,B=A_n,B_n$ (note that $A_n,B_n=X_1\dots X_L$ for all n). The conditional entropy is used to quantify the uncertainty about the classes of the collection of TS, given that T is fully or partially known. On the one hand, $H(C\mid A_n)$ consists of the amount of information required to predict the class labels, provided that the TS are known until the TP n. On the other hand, $H(C\mid A,B)$ corresponds to the amount of information needed to describe C, based on the knowledge of the entire T. The difference between these two conditional entropies measures the knowledge that the whole TS provides about the classes (correlation between C and $A,B$), which is not represented by the incomplete data (correlation between C and $A_n$). Thus, Equation (11) can be viewed as the lack of information caused by describing the structural relation C from $A_n$, i.e., the loss of knowledge for using the collection of TS only until the early TP in the classification process.

In addition to earliness in predicting the classes, the goal consists of finding the value n for which ${\mathcal{S}}_n$ represents the system S with reasonable complexity. Since this can be seen as a problem of learning the BN from Figure 1, both MDL and AIC scores are applied to the multivariate correlations for EC approach, in the interest of finding the best fitting model. These scores are used as two criteria for choosing the early TP, such that the selection of the model takes its simplicity into consideration. From Equation (3) and considering $P_{{\mathcal{S}}_n}$, described in Equation (10), the MDL score is defined as

$$\begin{array}{ccc}\hfill MDL(D\mid {\mathcal{S}}_n)& =& \frac{{log}_{2}N}{2}|{\mathcal{S}}_n\mid -{\displaystyle \sum _{i=1}^N}{log}_2\left[P_{{\mathcal{S}}_n}(A_n,B_n,C)\right]\hfill \\ & =& \frac{{log}_{2}N}{2}|{\mathcal{S}}_n\mid -{\displaystyle \sum _{i=1}^N}{log}_2\left[P(C\mid A_n)P(B_n\mid A_n)P\left(A_n\right)\right],\hfill \end{array}$$

(13)

where N is the number of instances in the dataset D, $|{\mathcal{S}}_n|$ denotes the number of independent parameters in the model, and $P_{{\mathcal{S}}_n}$ is the underlying distribution associated to the structure ${\mathcal{S}}_n$, which describes S as a representation of the given data. Similarly, the AIC score, applied to this context, is defined as:

$$\begin{array}{ccc}\hfill AIC(D\mid {\mathcal{S}}_n)& =& |{\mathcal{S}}_n\mid -{\displaystyle \sum _{i=1}^N}{log}_2\left[P_{{\mathcal{S}}_n}(A_n,B_n,C)\right]\hfill \\ & =& |{\mathcal{S}}_n\mid -{\displaystyle \sum _{i=1}^N}{log}_2\left[P(C\mid A_n)P(B_n\mid A_n)P\left(A_n\right)\right].\hfill \end{array}$$

(14)

As represented in the direct acyclic graph structure from Figure 1, the goal is to analyze how the structural relation $A_n$ is able to describe C, while the correlation between $B_n$ and C is neglected. For this reason, the computation of the network complexity only considers the relation between the early states and the class labels,

$$|{\mathcal{S}}_n|=|\left\{A_n,C\right\}|=||A_n||-1+\left(||C||-1\right)||A_n||=||A_n||\times ||C||-1,$$

(15)

where $||A_n||$ and $||C||$ denote the number of distinct observations in the structural relation $A_n$ and C, respectively. In Equations (13) and (14), the first term quantifies the complexity of the model, i.e., the amount of information required to encode not only ${\mathcal{S}}_n$, but also the data given ${\mathcal{S}}_n$. The second term measures the LL of the data based on the model, i.e., the amount of information needed to represent the dataset D according to the probability distribution $P_{{\mathcal{S}}_n}$. As n increases, the size of $A_n$ becomes larger, the number of correlations is higher and, consequently, the complexity of the model increases. In addition, the more information about the TS there is, the better the correlations describe the data, which means a decrease in the number of bits needed to describe C from $A_n$. The difference between these two terms describes the trade-off between the model complexity and the effectiveness of the fit to the data. The simplest model, that is able to use the least amount of correlations while maintaining a distribution as close to the original as possible, is found through minimizing both $MDL(D|{\mathcal{S}}_n)$ and $AIC(D|{\mathcal{S}}_n)$.

The Multivariate Correlations for Early Classification (MCEC) procedure, summarized in Algorithm 1, receives as input a comma-separated values (CSV) file, containing the TS and the respective class labels, and a scoring function $\varphi$. Both univariate and multivariate TS are allowed; however, the TS must be of fixed length. For both AIC and MDL, the overall time complexity of the MCEC algorithm is $O\left(L^3{N}^2{log}^2\left(N\right)\right)$, where L is the size of the time-series and N is the number of time-series in the data. Indeed, the procedure needs to store and count at most N configurations of $A_n$, and checking whether a configuration of $A_n$ (with size $O\left(L\right)$) already occurred takes $O(Llog(N\left)\right)$ time (using a binary search tree such as an AVL tree). Moreover, for each configuration of $A_n$ the procedure needs to store and count at most N configurations of $B_n$ (to establish the frequencies $P(B_n|A_n)$), which leads to an overall time complexity of $O\left(L^2{N}^2{log}^2\left(N\right)\right)$ for Step 3.

Algorithm 1 MCEC algorithm

1: for $n\in \left\{1,\dots ,L\right\}$ do 2:    Compute ${\mathcal{S}}_n=\{\{A_n,B_n\},\{A_n,C\}\}$ 3:    Compute $\varphi (D\mid {\mathcal{S}}_n)$ 4:    If $\varphi (D\mid {\mathcal{S}}_n)$ is minimum, store n 5: Output stored n

4. Experimental Results

The proposed algorithm was implemented in Java language, and is freely available in GitHub at https://joaopbeirao.github.io/MCECalgorithm/. In the interest of verifying the reliability of the proposed EC approach, an investigation on the accuracy of multiple classifiers was done. In this regard, we vary the length of the TS and check if the proposed cut-points attain a similar accuracy when compared with the complete TS. We note that the purpose of this assessment is not to elicit the best classifier among those used, but rather to understand the consequences of using a truncated time-series in classification tasks. Seven classifiers were considered, using the default parameters and stratified cross-validation with 10 folds: Naïve Bayes (NB), Bayes Net (BN), Sequential Minimal Optimization (SMO), C4.5 decision tree (J48), Reduces Error Pruning Tree (REPTree), Forest of Multiple Random Trees (RandomForest) and k-Nearest-Neighbor (kNN). We performed classification using the classifiers from the Weka Data Mining Software (version number 3.8) [25]. All the experiments were conducted using a PC with an Intel Core [email protected] GHz CPU and with 4 GB RAM.

From the MCEC algorithm, for each dataset, three values for the EC TP (n) were extracted. The first value is obtained from the difference in entropy measure: the smallest n, such that $H(C\mid A_n)-H(C\mid A,B)\le 0.3\times \left[H(C\mid A_1)-H(C\mid A,B)\right]$, which means that n corresponds to the TP where a reduction of $70\%$ from the initial value of entropy is verified, henceforth called $CH-70$, defined as the score:

$$CH-70(D|{\mathcal{S}}_n)=\left\{\begin{array}{cc}+\infty \hfill & \mathrm{if}H(C|A_n)-H(C|A,B)>0.7(H(C|A_1)-H(C|A,B))\hfill \\ n\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

The second and third values are the result of minimizing $MDL(D\mid {\mathcal{S}}_n)$ and $AIC(D\mid {\mathcal{S}}_n)$, respectively, i.e., n consists of the TP where the scores are minimum.

For analyzing the performance of the MCEC algorithm, we use thirteen benchmark datasets from the UEA & UCR Time Series Classification Repository [26,27]. This subset of examples is considered representative, as it comprises a diverse range of both dimensional parameters and classification conditions. Each dataset is composed of numeric TS with a fixed length, and their respective class labels.

For each example, a training set and a test set are provided separately. The preprocessing of the data included the aggregation of both training and test subsets in one single dataset. In addition, a TS discretization was performed following the guidelines proposed by Lin et al. [28] and Hu et al. [29]; no feature selection was performed. None of the datasets contained missing values; therefore, no imputation was required. The description of the data used in the experiments is given in

Table 1. Description of the benchmark time-series used in the experiments.

Dataset	#C	m	L	N	Type
ArrowHead	3	1	14	211	image
BirdChicken	2	1	25	40	image
Car	4	1	25	120	sensor
CBF	3	1	12	930	simulated
Coffee	2	1	71	56	spectro
Computers	2	1	5	500	device
ECG200	2	1	10	200	EGC
GunPoint	2	1	150	200	motion
Meat	3	1	7	120	spectro
SynthControl	6	1	6	8	simulated
Wafer	2	1	152	7164	sensor
ECG200M	2	2	13	200	ECG
WaferM	2	6	13	1195	sensor

. For each dataset, the number of class values (#C), the number of variables for each TP (m), the length of the discretized TS (L), the number of instances (N), and the type of data are provided.

The results from

Table 2. Experimental results of the MCEC algorithm.

Dataset	Earliness			Accuracy			Full
	$\mathit{CH}-\mathbf{70}$	$\mathit{MDL}$	$\mathit{AIC}$	$\mathit{CH}-\mathbf{70}$	$\mathit{MDL}$	$\mathit{AIC}$
ArrowHead	$57.14\%$	$7.14\%$	$42.86\%$	$72.51\%$	$38.39\%$	$70.14\%$	$78.20\%$
BirdChicken	$20.00\%$	$4.00\%$	$8.00\%$	$62.50\%$	$57.50\%$	$75.00\%$	$77.50\%$
Car	$52.00\%$	$12.00\%$	$40.00\%$	$70.83\%$	$40.83\%$	$70.83\%$	$65.83\%$
CBF	$66.67\%$	$33.33\%$	$66.67\%$	$91.61\%$	$74.09\%$	$91.61\%$	$94.95\%$
Coffee	$19.72\%$	$9.86\%$	$9.86\%$	$85.71\%$	$76.79\%$	$76.79\%$	$98.21\%$
Computers	$20\%$	$20\%$	$80\%$	$51.20\%$	$51.20\%$	$60.80\%$	$61.40\%$
ECG200	$50.00\%$	$20.00\%$	$30.00\%$	$87.50\%$	$76.50\%$	$87.50\%$	$85.50\%$
GunPoint	$24.00\%$	$0.67\%$	$15.33\%$	$92.00\%$	$71.50\%$	$83.00\%$	$99.50\%$
Meat	$28.57\%$	$28.57\%$	$57.14\%$	$69.00\%$	$69.00\%$	$74.50\%$	$75.50\%$
SynthControl	$62.50\%$	$25.00\%$	$50.00\%$	$89.50\%$	$62.67\%$	$85.00\%$	$92.33\%$
Wafer	$7.24\%$	$1.32\%$	$1.97\%$	$97.91\%$	$97.60\%$	$97.66\%$	$99.85\%$
ECG200M	$38.46\%$	$7.69\%$	$15.38\%$	$80.50\%$	$73.00\%$	$80.00\%$	$81.00\%$
WaferM	$30.77\%$	$7.69\%$	$30.77\%$	$93.22\%$	$89.36\%$	$93.22\%$	$96.15\%$

describe the MCEC algorithm effort in attempting EC, based on the analysis of the information contained in the datasets. For each dataset, it is given the Earliness as the percentage value associated with the EC TP n computed from MCEC algorithm: $\mathrm{Earliness}[\%]=\frac{n}{L}\times 100.$ The Accuracy columns present only the best (10-fold cross-validation) classification result (among all classifiers used) for the given data, predicting the class variable at TP n. The column Full contains the best outcome (among all classifiers used) for the complete TS, and it is used as a reference framework.

The results from the Earliness column of Table 2 confirm that in most datasets, it is viable to perform early classification. Concerning the $CH-70$ and $AIC$, the classification accuracy with fewer TPs outperforms the reference value (Full column) for the Car and ECG200 dataset. These examples suggest that it is possible to obtain a better classification performance using only part of the TS (in these cases, around 50% or even less) from the data. In the case of the Computers dataset, $AIC$ indicates the use of 80% of the TS, attaining with it an accuracy very close to the reference value. A similar result is obtained with the Meat dataset, where $AIC$ pointed out to use just over half the size of the TS. In the multivariate ECG200M dataset, $AIC$ pinpoint the use of only 15% of the MTS obtaining a difference of 1% less in accuracy, when compared with the reference value in the Full column. In other datasets (ArrowHead, BirdChicken, SynthControl, Wafer, and WaferM) accuracy outcomes with at most $-3\%$ in comparison with the full-length result are obtained. This means that, in these experiments, with fewer TPs analyzed (earlier in time), the loss in terms of classification accuracy can be diminished.

Regarding the scoring metrics, $MDL$ always proposes the lowest values for the early classification TP (Earliness column). However, the corresponding classification accuracy results of $CH-70$ and $AIC$ outperform the ones for $MDL$ in all cases. Moreover, $CH-70$ achieves higher classification accuracy in six out of thirteen cases, $AIC$ in three datasets, and in the remaining four instances, a draw is verified between the $CH-70$ and $AIC$. Therefore, from the experimental tests described in Table 2, in general, $CH-70$ achieves better results, in terms of classification accuracy, and $MDL$ demonstrates a superior earliness ability. $AIC$ evidences the foremost competence in balancing these two targets. Nevertheless, the early classification capabilities of the MCEC algorithm are acknowledged, seeing that this context is based on the trade-off between these two main objectives: accuracy and earliness.

4.1. Analysis of the ECG200 and Car Results

Herein, we detail the analysis of two datasets that illustrate the merits of the proposed method, complementing Table 2 with a graphical interpretation of the results, drawing conclusions both in terms of the relevant fraction of the time-series data for classification and the quality of the proposed scoring criteria.

The ECG200 data comprises two hundred time-series that traces the electrical activity recorded during one heartbeat used as part of R. Olszewski Ph.D. thesis [30]. It aims to discriminate between normal heartbeat and myocardial infarction. The original time-series has 96 TPs. After discretization, following the guidelines proposed by Hu et al. [29], the time-series was reduced to ten TPs. The results are depicted in Figure 2.

Results suggest that only a fraction of the heartbeat signal is crucial to discriminate between classes. In particular, $AIC$ (Figure 2a) and $CH-70$ (Figure 2c) proposed to cut the time-series at TPs 3 and 5, respectively; in $CH-70$ the cut point corresponds to the first n bellow the dashed red line. Indeed, both cut points proposed by AIC and $CH-70$ agree upon a peak in accuracy of 87.5% (c.f. Table 2), and so surpassing the full-length time-series in 2%. The $MDL$ criterion is more confident in this regard (c.f. Figure 2b), proposing to cut the time-series at $n=2$. However, in this particular case, it suggested a premature cut leading to lower prediction values, as depicted in Figure 2d.

In addition, the experimental results on the Car example are represented in Figure 3. Figure 3a,b represent the variation of $AIC(D\mid {\mathcal{S}}_n)$ and $MDL(D\mid {\mathcal{S}}_n)$, respectively, for $n\in \left\{1,\dots ,25\right\}$. While for $AIC$, the minimum is reached at $n=10$ (corresponding to 40% of 25 TPs), for $MDL$, the lowest value is attained at $n=3$ (corresponding to 12% of 25 TPs). In both cases, this extreme is followed by an irregular growth until $n=23$, where it stabilizes at a maximum value. Figure 3c describes the behaviour of $H(C\mid A_n)-H(C\mid A,B)$ while varying n from 1 to $L=25$. A decrease of $70\%$ from the initial entropy value is obtained at $n=13$ (corresponding to 52% of 25 TPs), depicted with a dashed red line.

Figure 3d includes the classification accuracy of the Car dataset. Note that there are two jumps in classification accuracy at $n=3$ and $n=10$, corresponding precisely to the timestamps elicited by the $MDL$ and $AIC$, respectively.

4.2. Statistical Significance

Experimental results were compared with statistical significance tests in order to understand the benefit of the trade-off between the two main goals in EC: accuracy and earliness. Among the tested datasets, the MCEC algorithm provided a value of n, with an associated percentage (Earliness). For each situation, the group of classifiers determined the Accuracy value. In addition, the classification of the full-data worked as a reference framework: no earliness and complete TS accuracy. In order to represent the balance between these two requirements, we consider the following quantity $BEA\left(p\right)=p\times (100-E)+(1-p)\times A$, where E and A correspond to the Earliness and Accuracy percentages, respectively, and p consists of the weight that determines the relevance given to each variable. Seeing that an accurate classification is desirable, as early as possible, $BEA$ describes the management of the two fundamental challenges of the EC problem. The thirteen datasets from Table 2 were considered, as well as their respective values of E and A, for each of the three measures that compose the MCEC algorithm, together with the reference framework. Note that all Full outcomes verify $E=0\%$, since the entire collection of TS is considered for classification.

Table 3. Comparison of the classification using the three MCEC algorithm cuts ($CH-70$, $AIC$ and $MDL$) and Full TS against each other, using the Wilcoxon signed-rank sum test applied to the trade-off experimental data, scored according to $BEA\left(p\right)$.

Comparison		p
		0	$\mathbf{0.25}$	$\mathbf{0.5}$	$\mathbf{0.75}$
$CH-70\iff MDL$	size	26	26	26	26
	p-value	$<0.01$	$<0.01$	$0.81$	$<0.01$
	better	⇐	⇐	→	⇒
$CH-70\iff AIC$	size	25	26	26	26
	p-value	$<0.01$	$0.01$	$0.34$	$<0.01$
	better	⇐	⇐	→	⇒
$CH-70\iff$ Full	size	26	26	26	26
	p-value	$<0.01$	$<0.01$	$<0.01$	$<0.01$
	better	⇒	⇐	⇐	⇐
$MDL\iff AIC$	size	20	20	20	20
	p-value	$<0.01$	$<0.01$	$0.23$	$0.37$
	better	⇒	⇒	→	←
$MDL\iff$ Full	size	26	26	26	26
	p-value	$<0.01$	$0.70$	$<0.01$	$<0.01$
	better	⇒	→	⇐	⇐
$AIC\iff$ Full	size	26	26	26	26
	p-value	$<0.01$	$0.37$	$<0.01$	$<0.01$
	better	⇒	←	⇐	⇐

includes the results of the Wilcoxon signed-rank sum test [31], for comparing the performance of classification using the MCEC algorithm timestamp n on the TS. These tests examine the relation between the scores in pairs, in order to verify if there is enough evidence to claim that the differences are significant, for a significance level of $\alpha =0.05$. The arrow in Table 3 points towards the measure with better performance, according to the value of $p\in \left\{0,0.25,0.5,0.75\right\}$. The double arrow means there is enough evidence to claim the difference is significant.

The results demonstrate that, for $p=0$, there is enough evidence to claim that Full surpasses all other measures. Furthermore, between $CH-70$ and the other two model selection criteria, the difference in entropy outperforms both scoring functions, and $AIC$ shows better results than $MDL$. All these differences are statistically significant. For $p=0.25$, $CH-70$ has the best performance in comparison with all of the remaining. The $AIC$ measure seems to achieve significantly superior results than $MDL$; however, there is not enough evidence to claim that $AIC$ outperforms Full, nor that the latter surpasses $MDL$. For $p=0.5$, the only assurance consists of Full performing the worst. Among $CH-70$, $MDL$, and $AIC$, the differences between them are not statistically significant. Lastly, at $p=0.75$, Full continues to be surpassed by all the others, as well as the difference in entropy in comparison with both model selection criteria. However, between $MDL$ and $AIC$, there is not enough evidence to confirm which performs the best.

5. Conclusions

This work proposes a novel algorithm, named MCEC, that aids in addressing the challenges associated with the task of early prediction in (univariate and multivariate) time-series data. Existing methods yield, for each time-series sample, a timestamp from which it is possible to perform early classification, failing to provide an overall data perspective. To the best of our knowledge, MCEC is the first approach that can grant the early opportunity of the entire data, allowing us to reason about prediction outcomes after understanding data idiosyncrasies.

MCEC is very flexible as it can be used with different scoring criteria, allowing for the trade-off between earliness and accuracy. We propose three measures: $CH-70$, $MDL$ and $AIC$. The achieved results confirm the ability of the MCEC method to examine the EC opportunity within a dataset. In general, the three criteria are capable of choosing a timestamp for which the time-series classification is plausible. Overall, the $CH-70$ obtains better accuracy results, $MDL$ demonstrates a superior tendency for earliness, and $AIC$ attains the most competent balance between both aims. Examples, where the earliness is very low, may indicate that, given the information available, the criterion recognized that the increase in the knowledge obtained from the data did not justify the growth in the model complexity required for its description. Conversely, the $AIC$ results demonstrate a more adventurous disposition in choosing the value for the early classification timestamp, which produced a relative success in benchmark data.

In terms of future work, several machine learning tasks can be developed based on the capabilities of this information-theoretic approach. A classification method can take profit from it, giving different attention to time-series whose classification timestamp deviates from that derived from the data; for instance, inspecting only part of electrocardiograph signals might improve existing classification methods [32]. It can also aid in determining the change point detection, as well as feature extraction and selection methods from multivariate time-series data. In the latter case, for instance, a greedy feature selection could be performed based not only on the difference in entropy measure but also on the model selection criteria. Finally, MCEC, if applied in a time-series from back/present to front/past, has the potential of unraveling the optimal Markov length of the stochastic process underlying the data.